Visually Grounded Commonsense Knowledge Acquisition

Large-scale commonsense knowledge bases empower a broad range of AI applications, where the automatic extraction (CKE) is fundamental and challenging problem. CKE from text known for suffering inherent sparsity reporting bias in text. Visual perception, on other hand, contains rich about real-world entities, e.g., (person, can_hold, bottle), which can serve as promising sources acquiring grounded knowledge. In this work, we present CLEVER, formulates distantly supervised multi-instance learning problem, models learn to summarize relations bag images an entity pair without any human annotation image instances. To address CLEVER leverages vision-language pre-training deep understanding each bag, selects informative instances via novel contrastive attention mechanism. Comprehensive experimental results held-out evaluation show that extract quality, outperforming pre-trained language model-based methods by 3.9 AUC 6.4 mAUC points. The predicted scores strong correlation with judgment 0.78 Spearman coefficient. Moreover, extracted also be into reasonable interpretability. data codes obtained at https://github.com/thunlp/CLEVER.